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Introduction 


A need within the NASA cost and schedule estimating community continues to be an ability to rapidly 
and parametrically assess the cost and schedules associated with maturing technologies through the 
Technology Readiness Level (TRL) scale. In response to this need, the Space Mission Analysis Branch 
(SMAB), in partnership with the HQ Cost Analysis Division (CAD) and the Game Changing 
Development Program Office (GCDPO), is working to further refine initial pathfinder efforts in this 
technology maturation cost and schedule estimating capability area. The end product will be a refined 
version of the Technology Cost and Schedule Estimating Tool (TCASE), associated data sets, and 
TCASE training available to personnel across the cost and technology assessment communities. 

TCASE is a technology cost and scheduled estimating tool originally developed under Task Order 
NNL12AC80T associated with Base Contract NNL11AA05B. The current research and development 
phase was focused on the following 4 subtasks: 

1 . Perform Data Research and Analysis 

2. Facilitate TCASE Use Case Studies 

3. TCASE Refinement 

4. Refine TCASE Training Materials and Provide On-Site Training 


This final report documents SpaceWorks’ contributions during the 12-month period beginning September 
2014 and ending September 2015. The following report sections will describe specific accomplishments 
and findings in each of the 4 subtask areas outlined above. The content of these sections is cumulative - 
spanning the entire contract year. Readers who have previously reviewed Quarterly Reports 1 


through 3 for this contract may wish to look for sections labeled 
contain new and previously unreported results and information. 


Quarter Update 


as these 


All other deliverable items required under Milestone 4 of the Task Order have been completed and 
submitted to the NASA point of contact. 
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Subtask #1: Perform Data Research and Analysis 

Analysis of TRL Step Transition Cost 

One of the fundamental use cases for TCASE is to help an analyst answer the question: What is the likely 
cost to mature (i.e. transition) a particular technology from a given starting TRL to a given ending TRL? 
The most straightforward way to use historical data to answer this question is to examine actual transition 
costs for executed projects that spanned the same range of TRL steps. Lortunately, data on the starting 
TRL, ending TRL, and associated development cost of a project record are some of the most readily 
available data in the TCASE database. 

Consider, however, that to make maximum use of the limited historical project data available, we would 
like to discretize the TRL transition data for a given project as much as possible to increase its 
applicability to future estimates. In other words, knowing that a particular historical project spent $10M to 
transition a technology from TRL 2 to TRL 4 is very useful; however knowing how much of that budget 
was associated with the TRL 2 to 3 step versus the TRL 3 to 4 step is even more desirable. Lortunately, 
such discrete data are available for a large number of records in the TCASE database. 

Note : During internal discussions, some TCASE team participants have voiced their distrust of the data 

associated with individual TRL steps within a project. The logic goes that whereas the starting 
and ending TRLs and overall budget of the project are hard to dispute, the timing and cost 
required for the individual steps may have been recorded less rigorously. Nonetheless, 
SpaceWorks has chosen to give these data the benefit of the doubt in this analysis. Our position is 
that even if some of these data are inaccurate, the high level trends across all projects may be 
revealing and useful. 

Upon inspection, of the 5,129 project records in the TCASE database, 851 records (~16%) contain 
detailed TRL transition cost data beyond simply the project start and end TRLs. Applying some data 
quality standards to these records, including requiring single step resolution (e.g. TRL 2 to 3, 3 to 4, etc.) 
and complete cost data, leaves 257 project data records that are suitable for transition analysis. 

SpaceWorks extracted transition cost data (all normalized to LY14 dollars) and corresponding TRL 
transition steps for each of these 257 records. We then did 2 types of analysis on these data: 

1. Calculated summary statistics (25 th /50 th /75 th percentile) across all 257 records by transition step 
and plotted these results. 

2. Segregated the data by Technology Area (TA) and repeated the analysis in (1). 

The results of these analyses can be seen in Ligures 1 and 2. Please note that SpaceWorks considers these 
analysis results to be preliminary at this time. The sample size is small, and further review and 
verification is warranted. 
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TRL Transition Step 


«^*25th Percentile ^^*Median ^^*75th Percentile 

Figure 1. Technology Readiness Level (TRL) Transition Cost versus TRL Step. 

[n = 257; data from NASA and other U.S. Government technology 
development projects conducted between 1994 and 2015] 



TRL Transition Step 

-^TA03 ^^TA04 TA08 TA12 

Figure 2. TRL Transition Cost versus TRL Step for Several Technology Areas (TA). 

[n = 87; Data from NASA and other U.S. Government technology 
development projects conducted between 1994 and 2015] 
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In Figure 1 , we observe a clear upward trend in the median transition cost at each subsequent TRL step. 
The data therefore suggest that, broadly speaking, the cost to mature a technology from TRL 5 to 6 will 
be greater than maturing the technology from 4 to 5, for example. To some readers, this trend may seem 
obvious or intuitive. Nonetheless, it is useful to examine the data and test such rule-of-thumb hypotheses. 
It is interesting to note that other researchers have examined similar datasets and come to similar 
conclusions. 1 


Figure 2 indicates a transition cost versus transition step trend that is less clear, however the variation 
between Technology Areas (TA) is plainly visible. Keep in mind that, compared with Figure 1, Figure 2 
relies on far fewer data points (n=87 vs n=257) and a higher degree of variability or noise is not 
surprising. Regarding the significance of TA, it is clear that certain TAs in the TCASE dataset (e.g. 
TA08) are associated with less costly TRL step transitions than are other TAs (e.g. TA04). Thus, it seems 
likely that any attempt to estimate the TRL transition cost of a technology project should take into 
account the nature of the technology itself - the purpose and type of the thing to be matured - in addition 
to the span of TRL steps to be covered. 


Analysis of TRL Step Transition Time 


4 th Quarter Update 


One notable event during this 4 th quarterly reporting period was the successful import of more than 2,000 
new project data records from NASA TechPort into the TCASE database. These new records nearly 
doubled the previous size of the database and made a positive contribution to the overall breadth and 
quality of data. As noted elsewhere, data records imported from TechPort do not include project 
budget/cost information - a fact that does somewhat limit their value to TCASE. However, project 
schedule information is typically included in TechPort records and, thus, the net effect of this large data 
import will be to bolster the analogy-based prediction of technology project schedule durations in 
TCASE. 


In order to understand high level project schedule duration trends, SpaceWorks conducted a data analysis 
on the TCASE database data to examine the time (schedule duration) required by the various projects to 
complete individual TRL step transitions. This analysis is similar to the approach outlined in the 
preceding section, “Analysis of TRL Step Transition Cost”. One aspect of the TRL step schedule duration 
analysis that differed from the cost analysis was that SpaceWorks had access to a larger sample size 
(n= 1,304). Again, schedule data is more plentiful in the database than cost data. The results of the TRL 
step transition time analysis are shown in Figure 3. 

The results suggest a positive correlation between TRL advancement level and transition time required. 
Specifically, advancing a technology from TRL 1 to 2, 2 to 3, or 3 to 4 appears to require about 6 months 
to 1 year per step. Once the technology has advanced to TRL 4, subsequent advances to TRL 5 and 6 
require about 2 years each in most cases. Of course these are broad trends across many technology areas 
and there are likely a number of factors influencing these trends. Not the least of which may be the 
technology development contracting systems in place at NASA and other government agencies that 
distribute funding in annual increments. The results of this analysis do not provide insight into the 


1 Evidence for Predictive Trends in TRL Transition Metrics, The Tauri Group, Prepared for NASA’s Technology 
Assessment and Integration Team, 2012. 
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sensitivity of these step transition times to funding level. For instance, would most technology 
advancements occur faster with more funding? 


3.5 



2 to 3 

3 to 4 

TRL Transition Step 

4 to 5 

25th Percentile 

^^“Median 

A 75th Percentile 


Figure 3. Technology Readiness Level (TRL) Transition Time versus TRL Step. 

[n = 1,304; Data from NASA and other U.S. Government technology 
development projects conducted between 1994 and 2015] 

Maintaining the TCASE Technology Project Database 

The value of the TCASE tool is broadly dependent on the quantity and quality of the technology 
development project data stored in the internal TCASE database. In this case, one of the most significant 
quality metrics is the age of the data. Recently updated project data is most desirable as it is thought to 
represent the current state of the art in NASA and U.S. Government technology development trends and 
practices. 

One approach to maintaining a set of current and valid data has been to periodically import project data 
from the NASA TechPort system. Compatibility between TCASE and TechPort has been inconsistent 
over the past couple of years. The root cause is the evolution of TechPort data formatting over time. In 
other words, the tables and fields of the TechPort database and the corresponding rows, columns, and 
worksheets in the Excel output have changed repeatedly. In this latest iteration, the TechPort project has 
begun including additional worksheets in the standard Excel export file, as well as re-ordering several 
columns of data across multiple sheets. It has been a challenge for those of us on the TCASE team to keep 
up with these changes as a 3 ld party. At this time, TCASE is compatible with TechPort and is capable of 
importing project data. Please see the section “Subtask #3: TCASE Refinement” for more details on how 
recent updates to TCASE should make maintaining compatibility with TechPort easier in the future. 
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Recent Expansion of the TCASE Technology Project Database 


4 th Quarter Update 


When SpaceWorks restored the functionality of the TechPort Importer in TCASE, the obvious next step 
was to download the latest set of TechPort project records and determine what - if anything - had been 
added to that database since the previous TCASE release (v0.93). In fact, it appears that 2,180 new 
project records have been entered into the TechPort system since approximately March 2015. The 
following paragraphs outline some of SpaceWorks’ observations about these new data and what impact 
they may have on TCASE estimating capabilities. 


As Figure 4 shows, the newly imported TechPort data predominantly represents technology development 
projects that were initiated in the past 5 calendar years. With this new data, the distribution of Project 
Start Year in the TCASE database became even more left skewed. SpaceWorks generally believes that 
reducing the average age of the data in the TCASE database tends to have a positive benefit on estimates 
generated by the tool. Policies and management practices for technology development processes evolve 
over time, and basing estimates on recently executed projects with contemporary management practices is 
desirable. 



Project Start Year 


Figure 4. Distribution of TCASE Database Entries by Project Start Year. 

The TCASE database has always included development projects from each of the 15 NASA Technology 
Areas (TA). Figure 5 shows how the recent addition of new TechPort data has altered the distribution of 
these TAs in the TCASE database. The most obvious change is the tremendous number of new data 
records added to TA08: Science Instruments, Observatories and Sensor Systems. Whereas TA08 
previously had an average to below average number of records compared with other TAs, it now has the 
largest number of records by far. Elsewhere, TAs 1 through 6, and 1 1 and 12 each have 200 to 500 
records. TAs 7, 9, 10, 13, 14, and 15 have relatively fewer records, although their number has increased 
from TCASE v0.93 to vl.0. Understanding the distribution of database entries (i.e. records) across the 
TAs is important for an estimator because the quantity of data in a given area drives the likelihood of 
finding a good analog or set of analogies to suit a new estimate. 
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New in TCASE vl.O 
TCASE v0.93 



6 7 8 9 10 11 12 13 14 15 

Technology Area 


Figure 5. Distribution of TCASE Database Entries by Technology Area. 

Finally, Figure 6 shows the distribution of TCASE database entries by Lead NASA Center for each 
project. Among other things, this graph indicates that the recently renamed Armstrong Flight Research 
Center (AFRC) is now listed as the Lead NASA Center on about 40 development projects. Also, the 
number of newly initiated development projects at each center has been proportional to the number of 
projects led by those centers in previous years. The exception is Goddard Space Flight Center (GSFC), 
which appears to have seen an increasing rate of new development projects in recent years. 



Lead NASA Center 


Figure 6. Distribution of TCASE Database Entries by Lead NASA Center. 
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Subtask #2: Facilitate TCASE Use Case Studies 


SpaceWorks worked closely with NASA Cost Analysis Division (CAD) personnel to plan, schedule, and 
carry out a TCASE pilot user (aka “beta tester”) session. The primary objective of the session was to 
continue to build a foundation for adoption of the TCASE analysis tool within the NASA estimating 
community. In order to support this objective, the pilot user session was designed to engage a small group 
of hand-picked NASA analysts, train them in the use of TCASE, present thought provoking example use 
case scenarios, and then solicit specific feedback. 

Preparation for the Pilot User Session 

Early planning for the session focused on identifying likely TCASE users within NASA and then 
extending an invitation to participate in the session. Mr. Scott May and Mr. Marc Greenberg at NASA 
CAD were instrumental in making this first step of the process a success. Over the course of several 
weeks, a list of participants was carefully assembled and invitations were prepared. 

Meanwhile, SpaceWorks was responsible for developing the training materials and use case examples that 
were presented during the session. In developing these examples, SpaceWorks attempted to capture the 
power and flexibility of TCASE while at the same time giving the session participants a chance to 
experiment and build intuition with the tool. In addition to the training materials and use cases, 
SpaceWorks also created a spreadsheet-based participant feedback form. 

Conducting the Pilot User Session 

The “live” pilot user session was held on November 19 th from 1 :30pm to 4:30pm EST and was led by Mr. 
Jon Wallace of SpaceWorks. Mr. Wallace was supported by other members of the TCASE development 
team including Mr. Mark Schaffer of SpaceWorks and Mr. May and Mr. Greenberg of NASA CAD. The 
session was conducted using the WebEx virtual meeting service and a telecon audio line. A total of 8 pilot 
users participated in all or part of the session: 

1. Pat Hunt (MSFC) 

2. Andy Prince (MSFC) 

3. Julie Williams-Byrd (LaRC) 

4. Tom Parkey (GRC) 

5. Param Nair (GSFC) 

6. Michael Johnson (GSFC) 

7. John Panek (GSFC) 

8. Susan Bertsch (JSC) 

The 3-hour session was interactive and included multiple questions and conversations. Most participants 
were able to remain online for the entire session. As the session drew to a close, Mr. Wallace reminded 
participants to complete and submit evaluation forms. The TCASE team ultimately collected 3 completed 
forms, and the results of these are summarized in the following section. 
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Pilot User Evaluation Results 

The TCASE Software and Training Evaluation Form contains questions that solicit both numerical rating 
answers and written response answers from the participants. The 19 numerical rating questions are 
divided into 7 evaluation categories ranging across the user experience. Table I below shows the averaged 
evaluation responses obtained from 3 participants. ( Note : Participants were asked to respond to each 
question with a rating from 1 =Poor to 5=Excellent.) 

Table I. Average TCASE Evaluation Ratings by Category (reported by 3 participants) 


1 Evaluation Form Categories and Questions 

Average Score | 

Access and Download 


4.3 

Ease of access to ONCE web portal 

4.3 


Ease of locating TCASE download link on ONCE web portal 

4.0 


Ease of downloading TCASE from ONCE web portal 

4.7 


Big Picture Items 


4.7 

Clarity of TCASE high-level purpose (why the tool was created) 

4.7 


Clarity of TCASE basic functionality (what the tool does) 

4.7 


Software Compatibility 


4.4 


Compatibility of TCASE with your primary work computer (works with your version of 
Excel, etc.) 

Ease of configuring Microsoft Excel to execute TCASE VBA macros (i.e. "Enable Macros") 
TCASE display and appearance on your primary work computer (appropriate screen width, 
font sizes, etc.) 


Software Training 4.7 

Sufficiency of TCASE project overview (depth and breadth of information provided) 4.3 

Clarity of example Use Cases in training documentation 4.7 

Value of "capstone" independent Use Case performed independently during training session 5.0 

Software Usability 3.9 

Clarity of TCASE input parameters (search terms, filters, weightings, etc.) 4.0 

Functionality of user interface objects such as buttons and pull-down menus 4.0 

Clarity of TCASE output information (statistics, plots, etc.) 3.7 

Software Documentation 4.0 

Completeness of TCASE User Manual 4.0 

Accuracy of TCASE User Manual 3.5 

Clarity of step-by-step instructions in manual 4.5 

Data Transparency and Traceability 4.3 

Transparency of TCASE source data 4.3 

Traceability from TCASE outputs back to source data 4.3 


4.3 

4.7 

4.3 
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The early results shown in Table I suggest that the TCASE team did an excellent job communicating the 
“big picture” reasoning behind the tool, and also organizing and conducting the user training that occurred 
during the pilot user session. However, the responses pointed to a need to clarify the output statistics that 
TCASE produces, including taking steps to ensure that the user manual describes these outputs well. 

The 5 written response questions are listed in Table II along with a sample of the responses received from 
3 participants. 


Table II. Written Response Questions and Sample Responses. 


1 Question 

Sample Responses i| 

What are the primary strengths 
of the TCASE analysis tool? 

Fast search capability, easy to use. 

It’s a very helpful search tool and database for putting together estimates by 
analogy. 

Simple interface. 

What are the primary 
weaknesses of the TCASE 
analysis tool? 

Needs more data, more information on the actual product of the technology 
development effort (i.e. did they test material coupons and catalog the 
information? Did they deliver a prototype detector?) 


I would prefer that there be an option to completely remove a data point, not 
just cross it out. It makes it slightly more visually challenging to pay attention 
to only the points you have selected as outputs. 

If TCASE had been available to 
you, would you have utilized the 
tool to support an estimating 
task in the past 12 months? 

No, mainly because we do not get asked to do estimates for technology 
developments. 

Most likely. 

Yes. 

Now that you are aware of 
TCASE, how likely are you to 
use the tool for an estimating 
task in the next 12 months? 

Hard to say, awareness of the tool may create demand. 

I will most likely take advantage of its search and database functions. 

Possible - analogy estimates of cost and duration for systems engineering 
support. 

Other comments 

This goes a long way to plugging the gap of low TRL technology estimating. 
Can you automate the updating of data from TechPort as a single click? 


Table II shows that session participants are generally optimistic about TCASE and the potential utility of 
the tool. The major strengths identified are consistent with the design objectives we have pursued in 
TCASE - simplicity and ease of use. The major weakness related to data quality is one that we are and 
have been aware of as a team. 
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Commentary on TCASE Use Case Studies 


4 th Quarter Update 


The TCASE pilot use case study has proven to be successful at achieving several goals. First, the session 
gave the TCASE research and development team a tangible deadline at which to have a functional 
software release and training materials ready. Without the use case study, the TCASE development 
timeline might not have experienced this accelerated drive to deliver a product. Second, whereas the 
workshops held during the first and second project years brought together a broad group from the 
estimating community, the use case study specifically engaged potential TCASE users/analysts. These 
individuals were able to provide feedback of the type only apparent to an end user of the software. 
Finally, the feedback obtained during the use case study led directly to bug fixes and feature 
enhancements that are now available in TCASE vl.O. These items are described in more detail in the 
following report section titled “Subtask #3: TCASE Refinement”. 
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Subtask #3: TCASE Refinement 

At the beginning of this contract year, the TCASE software tool was at version 0.93. Over the past 12 
months, SpaceWorks has enhanced and refined the tool in direct response to Pilot User Beta Test 
comments and to address various bug fixes and compatibility issues. The result is a new 1 .0 version of 
TCASE that is ready for release and has been provided to NASA as part of the final deliverable package 
for this contract. TCASE version 1 .0 finally dispenses with the “beta” tag that the tool has carried since its 
inception. 

UI Improvement - Toggle Analogies On/Off from Front End 

Based on feedback collected during the Pilot User Beta Test, SpaceWorks has added the capability to 
quickly turn analogy results on and off directly from the Front End worksheet. No longer does the user 
have to navigate to the Analogy Manager to perform this common task. Easy to read on/off toggles 
provide a quick visual status for each project. In addition, projects that have been turned off are also 
colored gray. This feature and its behavior are depicted in Figure 7 below. 

UI Improvement - Link Directly from Analogy Name to Data Viewer 

Also visible in Figure 7 is the new direct link capability from each analogy result to the Data Viewer 
worksheet. The link is formatted in a familiar hyperlink style (blue text with underline) in an effort to 
improve ease of use. This new link feature is quite useful for investigating a particular analogy result in 
greater detail before determining whether to keep or discard that item. The Data Viewer displays each and 
every field available for a technology project data record. 


Top Analogies (of 15 total analogies) 

On/Off Score 

ID 

Project Name 

TA 

Start 

Year 

TRL 

Start End 

© 








© 








© 

100% 

470 

Auburn University Center for SDace Power: En 

3.2.2 

1998 

4 

4 

© 

100% 

509 

Low Power Loss. Fail Safe MaeneticSusoensi 

3.2.2 

2001 

3 

4 

© 

100% 

533 

Aerosoace Flywheel TechnoloBV 

3.2.2 

1998 

4 

4 

© 

100% 

534 

Aerospace Flw^heel Technoloev (Enersv Stor< 

3.2.2 

1999 

2 

2 

0 

100% 

556 

Aerospace Flwi/heel TechnoloEv 

3.2.2 

1998 

4 

4 

© 

100% 

572 

Demonstration of Coordinated Momentum a 

3.2.2 

2001 

3 

3 

© 

100% 

696 

OPEN CORE ROTATOR 

3.2.2 

1998 


4 

© 

100% 

697 

Self-Bearins Motor-Generator for Flw^heels 

3.2.2 

2004 

2 

2 


Figure 7. Screenshot of Top Analogies section of TCASE vl.O Front End worksheet. 
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Refinements to TechPort Import Functionality 


4 th Quarter Update 


The NASA TechPort system has shown its value as a source of up-to-date technology project data for use 
in TCASE. Agency support for TechPort and the frequent updates and expansions of that system indicate 
that it will continue to be the go-to resource for this type of data for the foreseeable future. However, as 
discussed elsewhere in this report, the TCASE development team has been challenged throughout this 
year to keep in step with the “moving target” of TechPort data formats and organization. 


The first 2 iterations of the TechPort Importer worksheet in TCASE (in v0.93 and a prior beta release) 
were essentially hardwired for compatibility with the TechPort formatting that was current at the time of 
development. With the third iteration of the TechPort Importer found in TCASE vl.O, SpaceWorks has 
phased out the hardwired approach in favor of a more flexible design. Rather than use behind the scenes 
VBA code to map every TechPort parameter to its corresponding TCASE parameter, the new architecture 
uses a mapping worksheet to present an easily editable and customizable set of relationships to the user. 


If and when TechPort changes database fields or formats in the future, a TCASE user will simply need to 
reflect these changes on the TechPort Settings worksheet shown in Figure 8. Editing this worksheet 
requires no special knowledge of VBA or of the internal workings of TCASE. The process of editing the 
mapping worksheet is discussed further in the TCASE User Manual. 


SpaceWorks anticipates that this approach will be a long term solution to this recurring compatibility 
issue. 


Techport Settings 


Settings 

General Settings Technology Area 

Primary Worksheet Technology TA Worksheet “echnc ogy 

Data Header Row 6 TA Columns J,I,H,M,L,K # N 

Techport ID Col A 


TCASE Database Entry Techport Export Workbook Entry * Mandatory fields 


« 

Category 

Item 


Worksheet* 

Column(s)* 

Handler 

Ignore if Includes User Entry Override 

1 


Import File Name 

<- 

Technology 

A 

Write FileName 


2 

Contact Information 

Date Completed 






3 


Name 

<- 

All Attributes 

AE,AD,AC,AB,AA 

Use FirstEntry 

This item does not 

4 


Email 

<- 





5 


Phone 

«- 





6 


Role on Project 

<- 

All Attributes 

AE,ADAC,AB,AA 

Use FirstEntryHeader 

This item does not 

7 


Office 

«- 





8 


Organization 

<- 





9 

Project Description 

Project Title 

<- 

Technology 

C,B 

Use FirstEntry 


10 


NASA Program 

<- 

All Attributes 

Z 

Get NASAProgram 


11 


Start Date 


All Attributes 

AG 

Get_Date 


12 


End Date 


All Attributes 

AH 

Get Date 


13 


Start TRL 

<- 

Technology 

O 

Get Integer 


14 


End TRL 

«- 

Technology 

P,Q 

Use MaxVallnt 


15 


Total Cost (S) 

«- 

All Attributes 

BG 
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Figure 8. Screenshot of TechPort Settings worksheet in TCASE vl.O. 
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Detailed TCASE Change Log 

• v0.94 

- Front End 

■ Added ID and TA columns to Top Analogies section 

- Analogy Manager 

■ Fixed issue where additional columns of data were bring printed to right of data 
columns 

■ Rearranged columns to place Database ID near front of results 

- Classification Tree 

■ Hid calculations rows 20-75 


• v0.95 

- Front End 

■ Changed y-axis on System Hierarchy and Project Start Year from % to Count 

- TA Tables 

■ Updated TA nomenclature and added Aeronautics secondary TAs 


• v0.96 

- Database 

■ Corrected secondary and tertiary TA classification for numerous technologies 


• v0.97a 

- Calculations 

■ Fixed NASA Center and System Hierarchy chart counts to count both cost and 
schedule results 

- Data Exporter 

■ Corrected issue of missing "Data Sheet" worksheet template; added back into 
workbook 


• v0.97b 

- Visual Basic Routines 

■ Added error message feedback to all subroutines 

■ Correct issue where VBA was not properly calculating counts or percentiles 
when results had 0 cost or schedule data points (but >0 total data points) 


• v0.97c 

- Data Viewer 

■ Rebuilt Data Viewer to be driven by VBA rather than in-cell formulas 

■ Added "View" and "Clear" buttons to allow user to load and clear data in data 
viewer 
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■ Corrected data formatting for Start Date 


• v0.97d 

- Visual Basic Routines 

■ Added InflateCosts subroutine to inflate costs in database 

• v0.97e 

- General 

■ Updated cost labels to FY15 

- Database 

■ Updated all costs from FY14 to FY15 with inflation multiplier of 


• v0.97f 

- Front End 

■ Added capability for user to select to view only analogies with both cost and 
schedule data to Advanced Settings 


- Data Viewer 

■ Added message prompting user to enter Database ID if none has been entered 

- Database 

■ Corrected cost and date formatting for all cost and date data columns 


- Beta Release Version (2015-03-19) 

• v0.98a 

- Visual Basic Routines 

■ Corrected a bug in GenerateDi stri buti ons () that caused arrays to be 
improperly sized when only 1 analogy was returned by a search 


- Front End 

■ Added On/Off toggle to Top Analogies summary 

■ Added link from projects in Top Analogies summary to Data Viewer tab 

- Analogy Manager 

■ Added on-change events to On/Off toggles to automatically recalculate results 
when analogy is toggled on or off 

■ Removed 'Refresh Results' button 

- Calculations 

■ Added 'TotalCount' named range 

- Visual Basic Routines 
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■ Added supporting subroutines for Top Analogies changes 

■ Added supporting subroutines for Analogy Manager changes 

■ Added functionality to populate and clear 'TotalCount' named range 

■ Removed subroutine used by 'Refresh Results' button 


• v0.98f 

- Front End 

■ Removed unnecessary conditional formatting 

- Visual Basic Routines 

■ Fixed incorrect named range references in Include All and Exclude All routines 


4 th Quarter Update 


- General 

■ Added AFRC, WFF, and WSTF to NASA center lists throughout tool 

- Visual Basic Routines 

■ Added Techport Format worksheet for users to map TCASE data entries to 
exported Techport data 


4 th Quarter Update 


- Release Version (2015-09-16) 
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Subtask #4: Refine TCASE Training Materials and Provide On-site 

Training 

SpaceWorks devoted considerable time and effort to developing TCASE training materials during this 
contract year. It is somewhat difficult to distinguish work and accomplishments related to Subtask #2 
versus those associated with Subtask #4, since the training materials were developed to facilitate the pilot 
user testing. However, we will draw a distinction here between TCASE training and the TCASE pilot user 
session where the latter included training and other elements (feedback and evaluation process, ongoing 
communication, etc.). 

The full TCASE training package includes several elements, some of which were developed exclusively 
during this reporting period while others were pre-existing from earlier contract work in 2014. The 
training package includes: 

• Release version of TCASE software tool from ONCE 

• Release version of TCASE User Manual from ONCE 

• TCASE User Training presentation (new development) 

• TCASE Example Use Cases (new development) 

The TCASE User Training presentation is structured for an audience of new or novice users. The charts 
offer background material on the history and motivation behind TCASE, introduce the layout and 
contents of the tool, and present 3 example use cases that show rather than tell how the tool works. 

SpaceWorks carefully designed the use cases to encourage new users to keep an open mind regarding the 
range of questions that TCASE can help answer (rather than “pigeon hole” the tool based on a first 
impression). Feedback from the pilot user session indicates that the participants found the use cases 
beneficial - perhaps the most beneficial part of the entire session. The 3 use cases in the training 
presentation are: 

1. Estimate the cost and schedule duration for an advanced lithium ion battery tech maturation 
project with an initial TRL of 4 and an objective of attaining TRL 6. 

2. Estimate the schedule duration for an infrared detector tech maturation project with an initial TRL 
of 3 and an objective of attaining TRL 6 in a limited timeframe. Assume detector development 
must fit within the schedule of a fictional instrument development project. 

3. Estimate the range of cost associated with any type of tech maturation project that aims to 
advance from TRL 3 to TRL 5. Assume that a new fictional program office is being formed to 
manage such tech developments. What is the expected median cost to the program to maintain 10 
such technology projects? 50 projects? What is the expected 70 th percentile cost? 
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2014-15: Summary Conclusions and Recommendations 


4 th Quarter Update 


Data - Quality and Quantity 

During the 2014-2015 project year, the focus of the TCASE project has shifted from collection of 
historical data from many sources to securing a data pipeline between TCASE and NASA’s widely used 
TechPort system. TCASE vl.O implements a data import solution that was achievable within the project 
scope, while still providing the basis for a long-term ability to keep TCASE in sync with TechPort. 

Conclusion : TCASE data quantity is adequate and the established data pipeline will enable 

future growth. Data quality is now highly dependent the quality of data in 
TechPort. 

Recommendation : Technology development organizations within NASA should continue to work 

closely with project/program data tracking and archiving efforts (e.g. TechPort) 
to ensure that the right data is being captured at the appropriate quality level. 
TCASE would greatly benefit, for example, if project cost/budget information 
was included in TechPort in the future. 


New User Training and Use Cases 

SpaceWorks provided new user training to approximately 8-10 NASA civil servants and contractors 
during this contract year. Most of these individuals also participated directly in the pilot use case study 
conducted last year. Training materials in PowerPoint format have been provided as part of this final 
deliverable package. 

Conclusion : Both the training session and the pilot user study proved successful. With 

improvements that have been made in recent months, the training materials 
should remain useful to NASA for several years until significant external factors 
such as updates to Microsoft Excel® or changes to the agency’s technology 
tracking process render the package obsolete. 

Recommendation : User training materials for TCASE should be made available to new authorized 

users who obtain the tool via NASA ONCE. The training materials can support a 
self-directed learning experience. 

Software Maintenance 


TCASE vl.O is implemented as a macro-enabled Microsoft Excel® workbook. Development was 
performed using Excel 2010, although the tool has been tested on Excel 2007, 2013, and Excel for Mac 
2011. Future development - whether feature addition or bug fixes - is feasible as SpaceWorks has 
delivered the tool with unlocked, commented source code (VBA). 

Conclusion : Software tools must inevitably be updated to keep pace with changes to 

supporting technologies (e.g. OS) or user needs. TCASE vl.O has been delivered 
in a form that will enable NASA to make such future updates as desired. 
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Recommendation : Excel-based software tools offer many advantages, but also carry some 

disadvantages. One disadvantage that may apply to TCASE is the challenge of 
maintaining version control. Because TCASE is an unprotected workbook with 
unlocked VBA source code, any TCASE user could choose to modify his or her 
local copy of the tool. SpaceWorks understands that NASA maintains a number 
of Excel-based tools already, including the Project Cost Estimating Capability 
(PCEC). SpaceWorks recommends that NASA apply best practices and lessons 
learned from these similar tools to its maintenance of TCASE. 
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